Objective: For dose calculations in ion beam therapy, it is vital to accurately determine the relative stopping power (RSP) distribution within the treatment volume. A suitable imaging modality to achieve the required RSP accuracy is proton computed tomography (pCT), which usually uses a tracking system and a separate residual energy (or range) detector to directly measure the RSP distribution. This work investigates the potential of a novel pCT system based on a single detector technology, namely low gain avalanche detectors (LGADs). LGADs are fast 4D-tracking detectors, which can be used to simultaneously measure the particle position and time with precise timing and spatial resolution. In contrast to standard pCT systems, the residual energy is determined via a time-of-flight (TOF) measurement between different 4D-tracking stations. Approach: To show the potential of using 4D-tracking for proton imaging, we studied and optimized the design parameters for a realistic TOF-pCT system using Monte Carlo simulations. We calculated the RSP accuracy and RSP resolution inside the inserts of the CTP404 phantom and compared the results to a simulation of an ideal pCT system. Main results: After introducing a dedicated calibration procedure for the TOF calorimeter, RSP accuracies less than 0.6 % could be achieved. We also identified the design parameters with the strongest impact on the RSP resolution and proposed a strategy to further improve the image quality. Significance: This comprehensive study of the most important design aspects for a novel TOF-pCT system could help guide future hardware developments and, once implemented, improve the quality of treatment planning in ion beam therapy.