Conceptually, this is kind of a neat idea. Having looked at the pictures, it isn't completely crazy, but I think you'd find that performance as a rack will be lacking--most quality racks use welded junctures between the members so that the rack is a rigid body, which is important when you've got a load cantilevered out back. I think that it would be difficult to machine fixtures that would give you that same rigidity, and allow for quick demounting, and avoid damage to the poles. The fixtures might also wind up being heavy enough that you don't save much weight.
Tubus makes several racks that clock in under 1 lb, just as a point of reference.