Four-Way Ethics

So, you pull up to an intersection intending to go straight. There's a four-way stop and you are the last car to the intersection. The car that has been there the longest and therefore has the right of way is the car opposite from you and it is also going straight. The cars on either side have to wait for the car across from you to get all the way through the intersection. Is it ok to go across out of turn when that car goes? It doesn't make the others wait a single second longer, but at the same time, it wasn't your turn. Do the drivers in the cars beside you have the right to give you the hairy eyeball if you do go? Is it a matter of principle or a matter of consequence?