Natural palygorskite (Pal) nanorods have huge application prospect as a reinforcing filler of polymer material, but the presence of variable coloring metal ions (i.e., Fe(III)) in natural Pal may affect the comprehensive properties of polymer materials. In order to reveal the effects of removing coloring metal ions on the structure and mechanical, transparency and anti-aging performance of the polymer, a series of sodium alginate/Pal (SA/Pal) nanocomposite films were prepared using natural brick-red Pal (RPal) and white acid-leached Pal (HPal) as fillers, and their properties were investigated comparatively. Results indicate removal of partial metal ions in Pal greatly improved the mechanical, transmittance, water-resistance and anti-aging properties of SA/HPal film in contrast to the SA/RPal film, due to the intensified interaction between HPal and SA. After exposure in UV condition for 72 h, the SA/RPal film becomes brittle with a great decrease of tensile strength from 11.05 to 6.71 MPa; but the SA/HPal film still keeps good flexibility with only a slight decrease of tensile strength from 14.66 to 12.09 MPa, indicating a passive effect of removing Fe(III) on improving anti-aging performance of SA/Pal film. This research paves a theoretical foundation to extend the application of naturally abundant Pal in polymer materials.