Heirloom: This simply means that the variety comes from a stock that has been grown for a long time without any changes in its breeding and it's not one of the common commercial varieties that have been selectively bred for high yield, uniformity, mechanical harvesting, and storage-ripening desired of supermarket produce. Planting heirlooms in your garden helps protect genetic diversity, and they usually taste better, or store better, or grow better in a specific area since people wouldn't keep planting them in their food gardens year after year if they didn't (taste and reliability of their own food being more important than profit and ease of manufacture).
Most heirloom varieties are OP which means that they pollinate themselves via insects, wind, or animals and the seeds produced will most likely (but not always!) breed true (grow a plant identical to the parent). However, (because they are OP) in order to keep an heirloom pure, you have to make sure that it's either the only variety you plant, or that the seeds you save come only from fruits/veg that you know for sure didn't get cross-pollinated by another variety... otherwise you might end up with an F1.
OP - Open-Pollinated: This simply means that the plant variety is capable of reproducing through natural pollination methods, and it's seed should grow offspring that are identical to the parent (some exceptions exist*). OP varieties are important to homesteaders and preppers because it means that they can save the seeds from their own harvests and plant them year after year without having to buy new seed and can be reasonably assured that they'll get the same food from them. However, many (but not all) F1 and GM can reproduce through natural pollination methods but they don't normally bred true in successive generations.
It's important to understand the breeding characteristics of any species you plant. Most plants require both male and female flowers or flower parts to reproduce by pollination. However, some species don't have male and female flowers on the same plant, or the male and female flowers on the same plant won't pollinate themselves, in which case you'd need another plant of the same or similar species to pollinate them (many tree fruits are known to need a "surrogate" for pollination). And other species have both male and female flowers and will freely self-pollinate, there are even some species that are hermaphroditic (contain both male and female parts in the same flower) and they can also self-pollinate (even within the same flower!).
In addition to the male/female aspect, you also have to take into consideration how readily the cross-pollinates with other varieties (or species!). Most beans tend to be strongly inbreeding... they easily self-pollinate and don't usually cross with other varieties even if they are planted nearby. Tomatoes tend to be strongly outbreeding... they are easily pollinated by other varieties of tomatoes, and are genetically capable of crossing with other members of the nightshade family (peppers, eggplants & potatoes) although this is uncommon. Important to note that the fruit of a cross-bred plant is still the same fruit you'd expect... it's the fruit of the hybrid off-spring that will be different. If you have a strongly outbreeding variety in your garden and you want to keep it pure, either don't plant any other varieties, time your plantings so they flower at different times, physically distance them so crossing isn't likely (this is hard, some will cross-pollinate within 3 miles!), cover your rows and hand-pollinate, or bag a few blossoms to hand pollinate and only save the seeds of those fruits.
* Potatoes are one notable exception to the OP rule. Potato seeds produced by the pollinated flowers (as opposed to seed potatoes which are eyes from the tubers) do not bred true. In order to get successive generations of potatoes that are identical to the parent, you need to plant eyes from the potato tubers you collected from the parent. This is actually natural cloning, not really sexual reproduction.
F1 - Hybrids: Contrary to what some people believe, hybrids are not evil Frankenfood (GM). Hybrids are the result of intentional or unintentional cross-pollination of two parents of different varieties that create offspring with a combination of and/or slightly different characteristics than the parent. Doing this intentionally is a form of selective breeding (although only saving and planting seed from plants that have the characteristics you want is also selective breeding, just not crossing). There are advantages to planting F1s in your food garden, namely a phenomenon known as "hybrid vigor". Hybrid vigor means the first generation offspring of a cross grows better/faster, has higher yields, and/or is more resistant to pest/disease than the parent. This can be really beneficial to home food gardener who has less than perfect conditions and needs to grow a lot of food, or a type that maybe wouldn't do so well in their climate (super-early harvest, slow bolting, or extra cold-hardiness). Hybrids are completely natural, even if we humans had a hand in it, because they haven't crossed with anything that nature didn't already allow.
In some cases, intentional crossing results in F1 offspring that are sterile, which is sometimes a desired trait (seedless grapes and watermelons). In most cases, however, hybrids are genetically unstable, will not breed true, and subsequent generations will either become sterile or begin reverting to the parent or continue to mutate unpredictably. Sometimes repeated breeding of an F1 to itself, or to one of the parents, eventually creates an Fn (F2 is second generation, F3 is third, etc) that is stable and breeds true... and another reliable OP variety is born! Planting that seed will now consistently grow offspring with the expected characteristics.
GM - Genetically Modified: This is the evil Frankenfood (and, yes, it's one of my soapboxes). These are genetic mutations created in a laboratory by breaking into and fiddling with the DNA directly. These organisms could never be produced in nature because they combine genetic materials from entirely different, non-compatible, families and sometimes even cross kingdoms (plant with animal, etc). That's a bit of an oversimplification, but rest assured that these modifications (mutations!) are created by separating a gene responsible for a desired trait from the donor (usually a bacteria that may have been GM'd itself to produce it), breaking the DNA chain of the target gamete and inserting that gene into the chain where they think it'll produce the same desired result in the host. One thing we know about genes is that they're often responsible for more than one trait, and where you put them sometimes makes a big difference in what you end up with... which is what makes this practice a bit scary when you really think about it. What else is controlled by that gene, and what else is affected by that placement? Do you know? Do THEY? Do they even care as long as they turn a profit? Now you see why I'm a big fan of making farmers post their fields if they're planting GM crops, divulge the animals they're raising are GM (esp. if they're breeding them out!), and all GM foods being labeled in the market!
Unfortunately, many GM plants enjoy hybrid vigor almost as much (more in some cases) as their natural F1 hybrid counterparts. This means they may outperform other OP and F1 varieties, including escaping containment (becoming a weed) and cross-pollinating those varieties for several miles around where they are planted... which is a big problem if Farmer Jane is growing organic heirlooms down the road and suddenly gets a bunch of Frankenbastards next year (and gets sued by Monsanto for copyright infringement!). In some cases, the providers of GM seed alter them to be sterile or at least have a suicide gene that kills off the plant in some amount of subsequent generations, and the majority of them do not breed true. This means you can't save the seed and plant it next year (even if Monsanto won't sue you for copyright infringement and licensing violation). However, there's really no telling whether or not the sterility and/or suicide switch will actually work as intended if that plant is actively cross-pollinating with other natural varieties in the wild because Nature is nothing if not fickle and resilient. (You all remember Jurassic Park and the frog DNA, right?) Even if the safeguards do function, we've now possibly corrupted, and effectively destroyed, the genetic purity and diversity of any crop which it cross-pollinated within several miles... and several miles of those... and several miles of those... until it's finally noticed. Mutations can be surprisingly aggressive that way. If you do decide to plant a GM in your garden, please do everyone a favor and pinch off anything that even looks like a flower bud before it opens! Or at least don't move anywhere near me!
Edited to Add: Gungnir just informed me that it seems someone in our government actually showed some rare forethought and made engineering a "suicide", or terminator, gene into a GMO illegal because of the "remote possibility" that the organism would escape containment, cross-breed with other varieties in the wild, and effectively decimate the entire population of that organism. So no terminator genes (that we know of, at least... corporations can do some pretty sneaky things to protect their profits and market advantage sometimes).
Now, I know there are all sorts of "good reasons" to GM food... resistance to herbicides (Roundup Ready), built-in pest resistance (Bt bred), and additional nutrients or quantity of nutrients (Golden Rice). All of these, on the surface at least, appear to be good, humanitarian advantages allowing farmers to produce more and more nutritious foods more easily for more people. I'm all for feeding the people... but doing a wrong thing for the right reasons... what do they say about the road to Hell and it's paving? In most cases, the things that these crops are being GM'd for are a direct result of our own arrogant adherence to unnatural and unsustainable agricultural practices. Many plant diseases, pest and weed pressures can be addressed through natural F1 hybrids and instituting healthier management practices like crop rotation and fallow fielding with green manure groups, and crop diversity rather than mono-cropping. And let's not forget that a lot of the countries that these "humanitarian" GM crops are supposed to help feed don't even want it because it's GM and current trials aren't going as well as promised!
So, to wrap up:
- Get heirlooms if you like them, but be considerate and try to maintain their genetic purity for the sake of diversity. Before spending the extra money, make sure that it will actually grow well in your conditions and produce the yields and flavors you're looking for... just because it's heirloom doesn't automatically mean it's better.
- Get OP if you want to save seeds and plant again next year to get the same crop. Doesn't matter if it's heirloom or one of the newer varieties as long as it grows well in your conditions and gives you a harvest that meets your needs and expectations in both yield and flavor.
- Get F1 if you have difficult conditions to contend with and you don't intend to save the seed to plant next year, just be careful that it doesn't cross with your other OP's & heirlooms. If you feel adventurous, you can even carefully attempt crossing some of your own varieties to get the hybrid benefits.
- Avoid GM if at all possible... like if it's the only viable seed left to plant on the entire planet and there are absolutely no other sources of food available.