Jim Caldwell won a gold medal at the 1961 National AAU boxing finals in the 139 lbs. division in Pocatello, ID. He joined the U.S. team on a European tour to compete against outstanding amateur boxers. He boxed in England, Ireland, and Scotland.

At age 21 he compiled a 32-6 amateur record against top national competition. He won four straight district Golden Gloves championships on the state level. He participated in the national Golden Gloves finals for three years and was a silver medalist in 1961.

Caldwell began boxing at aged 17 for the Menominee Boxing Club under the tutelage of 2026 NAIAHF inductee Alex Askenette Sr.

He was named outstanding novice boxer at the 1958 Fond du Lac (WI) district Golden Gloves, outstanding fighter at the 1960 Rockford (IL) tournament, and named winner of the Barney Rose sportsmanship trophy at the 1961 national Golden Gloves tournament.