Kemp dominated the closing overs to score his runs off 64 balls after South Africa's middle-order had rallied from 85 for four to 182 for six.
Schedule: New Zealand in South Africa
Ashwell Prince [ Images ] added 31, with five boundaries, and Mark Boucher [ Images ] 33, but Kemp led the way as the hosts notched up 115 runs in the last 15 overs to reach their winning target of 250.
Kemp started his innings patiently before changing his approach to take 20 runs off Nathan Astle's last four balls of his sixth over.
After he and Boucher put on 65 for the sixth wicket off 69 balls, the South African wicketkeeper missed a wild sweep at Daniel Vettori [ Images ] and was bowled.
Shaun Pollock [ Images ] came in and blasted 37 off 29 balls to set the hosts on to the victory path.
The New Zealand bowlers had earlier stifled the South African top-order, with the normally free-scoring Herschelle Gibbs [ Images ] failing to hit a boundary in his contribution of 25 off 69 balls.
Shane Bond [ Images ] began erratically by bowling numerous wides in his opening over but struck back in his second when he trapped Boeta Dippenaar (10) in front of his stumps.
The collapse left South Africa struggling on 85-4 in the 21st over before Kemp and Pollock came to the rescue.
With Kemp and Pollock both caught on the boundary off Jeetan Patel's [ Images ] bowling in the penultimate over, Nicky Boje hit the winning runs.
He skied Bond over mid-on and Stephen Fleming [ Images ], running back, was unable to cling on to an awkward catch coming over his shoulder.
Craig McMillan [ Images ] had earlier scored a composed half-century to steer New Zealand to 249 for eight after they won the toss and chose to bat.
McMillan worked the ball around sensibly on a surprisingly slow pitch to compile 66 in 75 balls.
He and Jacob Oram [ Images ] put on 75 for the fifth wicket to deny a South African attack that dominated early on.
Astle (1) and Lou Vincent (8) were dismissed in the first six overs, offering easy catches to Boucher off Pollock and Makhaya Ntini [ Images ] respectively.
Kiwi captain Fleming hit a brisk 45 to lift his team to 70 for three before falling to a fine diving catch by Gibbs at backward point off Andre Nel [ Images ].
When Kallis trapped Hamish Marshall lbw for 16, New Zealand had slipped to 97 for four in the 21st over.
Once the pacemen came off, McMillan and Oram flourished against the slower bowling of Kallis and spinner Boje.
McMillan was dismissed in the 43rd over when he tried to chip Kallis over the infield and lobbed a catch to Prince at wide mid-on, giving South Africa's premier all rounder his 200th one-day wicket in his 221st international.
Kallis is the third South African to reach the landmark, after Pollock and Allan Donald. He is also the eighth player to score 2,000 runs and take 200 wickets in one-day internationals.
Oram's fine supporting innings ended when he was run out by skipper Smith for 35 after McMillan had turned down a quick single.
Nel picked up three for 42 while Kallis took two for 45.
New Zealand Innings
S. Fleming c Gibbs b Nel 45
N. Astle c Boucher b Pollock 1
L. Vincent c Boucher b Ntini 8
H. Marshall lbw b Kallis 16
C. McMillan c Prince b Kallis 66
J. Oram run out 35
B. McCullum c Dippenaar b Nel 28
D. Vettori c Prince b Nel 1
A. Adams not out 30
K. Mills not out 1
Extras (b-1 lb-11 nb-2 w-4) 18
Total (for 8 wickets, 50 overs) 249
Fall of wickets: 1-7 2-18 3-70 4-97 5-172 6-200 7-204 8-241
Did not bat: S. Bond
S. Pollock 10 - 2 - 43 - 1
M. Ntini 4 - 0 - 19 - 1 (nb-2)
C. Langeveldt 10 - 1 - 48 - 0 (w-1)
A. Nel 9 - 0 - 42 - 3
J. Kallis 9 - 1 - 45 - 2 (w-2)
N. Boje 6 - 0 - 32 - 0 (w-1)
J. Kemp 2 - 0 - 8 - 0
South Africa Innings
G. Smith b Adams 19
B. Dippenaar lbw b Bond 10
J. Kallis b Mills 0
H. Gibbs c H. Marshall b Bond 25
A. Prince b Astle 31
M. Boucher b Vettori 33
J. Kemp c Vincent b Patel 73
S. Pollock c Adams b Patel 37
C. Langeveldt not out 0
N. Boje not out 3
Extras (b-1 lb-4 nb-1 w-13) 19
Total (for 8 wickets, 49.3 overs) 250
Fall of wickets: 1-29 2-34 3-41 4-85 5-117 6-182 7-247 8-247
Did not bat: A. Nel
S. Bond 9.3 - 1 - 50 - 2 (w-2)
K. Mills 10 - 2 - 38 - 1
A. Adams 6 - 0 - 28 - 1
D. Vettori 10 - 0 - 35 - 1 (nb-1)
N. Astle 6 - 0 - 46 - 1 (w-1)
J. Patel 8 - 0 - 48 - 2 (w-1)
Result: South Africa won by 2 wickets