Address comments

Author: icexelloss

doc/source/whatsnew/v0.24.0.txt

@@ -1210,6 +1210,7 @@ Indexing
 - :class:`Index` no longer mangles ``None``, ``NaN`` and ``NaT``, i.e. they are treated as three different keys. However, for numeric Index all three are still coerced to a ``NaN`` (:issue:`22332`)
 - Bug in `scalar in Index` if scalar is a float while the ``Index`` is of integer dtype (:issue:`22085`)
 - Bug in `MultiIndex.set_levels` when levels value is not subscriptable (:issue:`23273`)
+- Bug where setting a timedelta column by ``Index`` causes it to be casted to double, and therefore lose precision (:issue:`23511`)
 
 Missing
 ^^^^^^^

pandas/tests/indexing/test_timedelta.py

@@ -81,15 +81,17 @@ def test_numpy_timedelta_scalar_indexing(self, start, stop,
         expected = s.iloc[expected_slice]
         tm.assert_series_equal(result, expected)
 
-    def test_set_dataframe_column_by_index(self):
-
+    def test_roundtrip_thru_setitem(self):
+        # PR 23462
         dt1 = pd.Timedelta(0)
         dt2 = pd.Timedelta(28767471428571405)
-
         df = pd.DataFrame({'dt': pd.Series([dt1, dt2])})
+        df_copy = df.copy()
         s = pd.Series([dt1])
-        value_before = df['dt'].iloc[1].value
+
+        expected = df['dt'].iloc[1].value
         df.loc[[True, False]] = s
-        value_after = df['dt'].iloc[1].value
+        result = df['dt'].iloc[1].value
 
-        assert value_before == value_after
+        assert expected == result
+        tm.assert_frame_equal(df, df_copy)