oyle, New York, one of the world's foremost auctioneers and appraisers, made Hollywood history on Sept. 27 as property from the estate of James Cagney, one of Tinseltown's most legendary stars, went on the auction block.   The highlight of the sale was the 1942 Academy Award he won for Best Actor for his role in the musical Yankee Doodle Dandy. The coveted Oscar was sold for $170,000 U.S. Other items included his 1961 Bentley, which fetched $56,000 U.S.   The spectacular diamond engagement ring Cagney gave to his wife Frances was a steal at $30,000 U.S. and other personal property included paintings by Cagney and prominent American artists.