The breeding blanket does slow the neutrons, but there needs to be a first wall material that does not ablate into the plasma. You need very little of non-hydrogen material in the vacuum to cause a density collapse. If you made the wall out of liquid lithium then there would be a lot of lithium in the plasma.

Tungsten is a good choice for a first wall material because of its uniquely high melting point, low rate of embrittlement in high neutron flux, and short-lived radioisotopes.