Make sure that the physical volume isn't used by any logical volumes by using then 'pvdisplay' command:
| 
# pvdisplay /dev/hda1
--- Physical volume ---
PV Name               /dev/hda1
VG Name               myvg
PV Size               1.95 GB / NOT usable 4 MB [LVM: 122 KB]
PV#                   1
PV Status             available
Allocatable           yes (but full)
Cur LV                1
PE Size (KByte)       4096
Total PE              499
Free PE               0
Allocated PE          499
PV UUID               Sd44tK-9IRw-SrMC-MOkn-76iP-iftz-OVSen7
         | 
If the physical volume is still used you will have to migrate the data to another physical volume using pvmove.
Then use 'vgreduce' to remove the physical volume:
| 
# vgreduce my_volume_group /dev/hda1
         |