Assuming one has the proper bit - is there a good reason not to mill material (aluminum say) on a drill press by moving the material around? Seemingly they do more or less the same (rotary motion)
The thing that a traditional milling machine does well is handle sideways loads. Your drill press is designed for up/down loads.
Assuming you get the bit speed right, the downside to your plan is premature wear of the bearings on the drill press.
Now, if you can use it to only cut down into the stock, then you should do okay (though slow). (Assuming adequate safety/clamping of stock/etc.)
Yes you can. I've used a 1/4" milling bit to make a hole of diminishing triangular cross-section in 1" solid brass rod, so as to adapt a corn mill rotor to a slow, high torque DC motor. That said, doing such work is dangerous. One slip, and several of your fingers are toast. As Aloysius Defenestrate said, it's also not very good for the bearings in your drill press.
Aluminum, depending on the alloy, tends to be a little grabbier than brass. If you're holding things by hand that turns out to be a bad thing.
You might also consider using a small scale rotary tool (example). They're not nearly so torquey as a full scale drill press.
Usually the chuck+arbor is held in by a taper (the arbor would have two tapers on it). Side loads can dislodge the taper and the chuck (and usually the arbor) falls out, causing general mayhem not to mention personal danger.
Folks have been known to glue the taper in place.
From a Grizzly manual:
A drill press does not have bearings designed for side load and it is nowhere near as rigid as a proper milling machine so you can expect chattering and possibly broken end mills (especially if you use smallish solid carbide tools). Backlash in a crappy x-y vise can cause the tool to be pulled into the workpiece, especially with conventional vs. climb milling.
Typical SFM for (say) 6061 aluminum is around 280 so the spindle RPM will be a bit slow for end mills smaller than around 0.5" diameter. You can compensate by feeding slower, but typically small mills don't turn that much faster than drill presses using similar diameter tools in similar materials (as you might expect).
It's ugly. The drill press has way too much lash.
I mean, try it. Take a piece of aluminum, bang a dimple into it with a center punch, and then intentionally try to drill a hole 1/16" away. No matter how hard you try, the drill bit will sidestep into the center of the dimple. That's how much lash it has, and it's there on purpose, to do that.
And you know how side-lash is on a mill, the lash comes out and you hit a hard point. Not on a drill press; side-lash stays boingy until something breaks. At best, this will invite the bit to chatter.
Cutting is all about feeds and speeds. Drill presses do not have tables, so you cannot use the table to assure uniform feed. You won't get consistent feeds trying to hold the work by hand, so you'll gall and dull your endmills, if not break them, while work-hardening the material, and the lash will allow the bit to weave all over in search of the softest material.
Consider your options for getting time on a real mill.
Thanks to the Maker movement, there are now hundreds of places sorta like gym memberships but with machine tools... I don't know exactly what to call them. Such as TechShop but there are many others. Or community colleges now more open to public access. Don't be bashful, swing by and they'll give you the grand tour for free.
Other options are online machine shops or hiring a local machine shop.
I would second Sphehro Phfhany's comment: I have used cheap mill-drill machines in the past with only a taper arbor to hold the chuck into the drill quill. Invariably the vibrations from the mill cutter and the fact that the cutter itself is trying to pull 'down' due to the flutes in the cutter, it will dislodge the entire chuck from the machine. I've never been injured by this (fortunately) because it usually throws it downwards against the top of the work-piece, but it often means the job is ruined. I've tried putting a piece of paper in there (an old machinist's trick for taper arbors on lathes) but nothing helps, short of gluing it in as mentioned above. 'real' milling machine arbors have a threaded hole in the end. There is a threaded draw-bar which goes into this hole to hold the taper arbor in place. Photo milling machine Taper Arbors The arbors in that photo also have a slot cut in the collar at the bottom of the taper. This engages two drive pins mounted in the quill nose: with the draw-bar holding them in and the drive pins in those slots, its impossible for the arbor + chuck to slip.
If you know someone with a lathe you might be able to have them make up a new (softer) taper with a threaded hole in the end like this, and then take a big piece of threaded rod, pass it down through the hole in the top of the drill press (assuming a hollow quill) and then put a big nut and washer on the top to act as a draw-bar.
All that said, about 99% of the Chinese hobby mill drills and combo lathe + mills just have a standard chuck with no draw-bar so doing your own conversion shouldn't be any more 'dangerous' than using a commercial (if badly designed) machine.
None of this will help with the bearings of course, but if you are only milling light material it should suffice: the important part is not to have a machine which hurls spinning cutters at random intervals...
The other consideration is most drill presses have a round column (the big shaft at the back that holds the whole machine together) and the worktable is just clamped around this column with no way to actually stop the table rotating around it. The side-loads from milling even aluminium can cause either the table or the whole head to rotate around the column, which obviously destroys any accuracy in your job. There have been many articles in modelling magazines like Model Engineer about fixing this both on cheap mills and converted drill presses.
TL-DR: It's entirely possible to convert a drill press into a mill, but it takes a fair bit of work and will never be as rigid as a real mill. You're better of buying a real mill, and by that I don't mean a cheap hobby mill which looks like just a drill press with an X-Y table: a square column (or dovetail slides) and a proper locking arbor are essential.