It is generally stated on the basis of the chronicler Jan Długosz that the castle was founded in the 15th century on the initiative of bishop Wojciech Jastrzębec. However, the results of archaeological research and source exploration allow to move the date of the beginning of the building to the XIV century. In 1397, the castle was the subject of a dispute between the castellan Klemens Mokrski, and Małgorzata, widow of Paszek from Rytwian. In 1414 Rytwiany were already in the hands of the nephews of Wojciech Jastrzębiec. The castle, rebuilt by the bishop, almost until the end of the 15th century was the family residence of Jastrzębiec-Rytwian. In 1516, a fire broke out in the castle. Another destruction came in 1657 with the Rakoczi army. Although heavily ruined, the castle was still used until the beginning of the 19th century. It was not until 1859 that the next owners, the Potocki family, dismantled most of the walls of the castle.
In the first phase of development from the end of the fourteenth century, a one-bay, three-storey building of 13×39 meters was probably erected. Probably to the north of it, on the other side of the courtyard stood an analogous, parallel house. In the second phase of the first half of the 15th century, two houses site was surrounded by defense walls on the rectangular plan. A main house was erected in the northern curtain. The gate was probably in the eastern part of the wall.
To this day has survived only a fragment of brick wall with slanting buttress, decorated with a grid of diamonds and frieze of sloping bricks.
Leksykon zamków w Polsce, L.Kajzer, S.Kołodziejski, J.Salm, Warszawa 2003.
Wróblewski S., Zamki i dwory obronne województwa sandomierskiego w średniowieczu, Nowy Sącz 2006.