How to validate recommender model in healthcare?












0












$begingroup$


In order to validate a recommender model, a usual approach is create a hold-out set that will provide random suggestions (similar to an A/B testing setup).
However, in healthcare applications, this cannot be possible as a random suggestion can put at risk a patient's life.
Hence, what is a reasonable approach to validate the model?






recommender-system data-product healthcare







share|improve this question











$endgroup$












  • $begingroup$
    Could you provide a little bit more detail about what sort of work you're doing? I'm assuming a lot, like that the randomness relates to group assignment and not the type of treatment itself, but there isn't much detail here.
    $endgroup$
    – Upper_Case
    6 hours ago
















0












$begingroup$


In order to validate a recommender model, a usual approach is create a hold-out set that will provide random suggestions (similar to an A/B testing setup).
However, in healthcare applications, this cannot be possible as a random suggestion can put at risk a patient's life.
Hence, what is a reasonable approach to validate the model?






recommender-system data-product healthcare







share|improve this question











$endgroup$












  • $begingroup$
    Could you provide a little bit more detail about what sort of work you're doing? I'm assuming a lot, like that the randomness relates to group assignment and not the type of treatment itself, but there isn't much detail here.
    $endgroup$
    – Upper_Case
    6 hours ago














0












0








0





$begingroup$


In order to validate a recommender model, a usual approach is create a hold-out set that will provide random suggestions (similar to an A/B testing setup).
However, in healthcare applications, this cannot be possible as a random suggestion can put at risk a patient's life.
Hence, what is a reasonable approach to validate the model?






recommender-system data-product healthcare







share|improve this question











$endgroup$




In order to validate a recommender model, a usual approach is create a hold-out set that will provide random suggestions (similar to an A/B testing setup).
However, in healthcare applications, this cannot be possible as a random suggestion can put at risk a patient's life.
Hence, what is a reasonable approach to validate the model?






recommender-system data-product healthcare




recommender-system data-product healthcare






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited 6 hours ago









Brian Spiering

4,2581129




4,2581129










asked 7 hours ago









tashuhkatashuhka

356310




356310












  • $begingroup$
    Could you provide a little bit more detail about what sort of work you're doing? I'm assuming a lot, like that the randomness relates to group assignment and not the type of treatment itself, but there isn't much detail here.
    $endgroup$
    – Upper_Case
    6 hours ago


















  • $begingroup$
    Could you provide a little bit more detail about what sort of work you're doing? I'm assuming a lot, like that the randomness relates to group assignment and not the type of treatment itself, but there isn't much detail here.
    $endgroup$
    – Upper_Case
    6 hours ago
















$begingroup$
Could you provide a little bit more detail about what sort of work you're doing? I'm assuming a lot, like that the randomness relates to group assignment and not the type of treatment itself, but there isn't much detail here.
$endgroup$
– Upper_Case
6 hours ago




$begingroup$
Could you provide a little bit more detail about what sort of work you're doing? I'm assuming a lot, like that the randomness relates to group assignment and not the type of treatment itself, but there isn't much detail here.
$endgroup$
– Upper_Case
6 hours ago










1 Answer
1






active

oldest

votes


















0












$begingroup$

You should still be able to use a validation set to evaluate the model, whether or not you pursue an experimental approach. (Specific features of your model and investigations may tweak these, but this is based on what's already been posted alone).



There is nothing wrong with A/B group assignment and testing in a medical context, with a few caveats (this list is not exhaustive):




  • The relevant clinical/medical knowledge must be in a state of
    equipoise (it's not already clear that one approach is better than
    another, or which is better is genuinely not known).

  • Individuals should be aware that they are participating in a study, and that they are being routed to
    group A or B, and have the option to decline their assignment (or,
    conversely, they have been made aware of the experimental assignment
    and have consented to participate in advance).

  • An institutional review board should evaluate your proposed
    experiment and signed off on it. This, of course, presupposes that
    you have access to such a board composed of members able to make
    those assessments.


Those can be a tall order, but you don't necessarily have to perform a prospective, double-blind experimental study in order to glean some information. A retrospective study could provide some insight as well, and your process for the validation set would be something like:




  1. Prepare your recommender model

  2. Feed your data through the model, without looking at actual outcomes

  3. Match your model output to actual outcomes to see whether or not
    people followed the recommendation (whether they ever saw that
    recommendation or not)

  4. Compare the results of people that ended up going with each
    recommended approach (A vs. B), as well as those who "followed" the
    recommendations or not (Recommended-A-did-A vs. Recommended-A-did-B,
    etc.)


Retrospective studies are generally not as good as well-designed, well-executed prospective experimental studies, but they can still provide a lot of information. In situations where prospective experimentation is impossible or undesirable, the information a retrospective study provides may be the very best you can actually get.






share|improve this answer









$endgroup$














    Your Answer





    StackExchange.ifUsing("editor", function () {
    return StackExchange.using("mathjaxEditing", function () {
    StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
    StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
    });
    });
    }, "mathjax-editing");

    StackExchange.ready(function() {
    var channelOptions = {
    tags: "".split(" "),
    id: "557"
    };
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function() {
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled) {
    StackExchange.using("snippets", function() {
    createEditor();
    });
    }
    else {
    createEditor();
    }
    });

    function createEditor() {
    StackExchange.prepareEditor({
    heartbeatType: 'answer',
    autoActivateHeartbeat: false,
    convertImagesToLinks: false,
    noModals: true,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: null,
    bindNavPrevention: true,
    postfix: "",
    imageUploader: {
    brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
    contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
    allowUrls: true
    },
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    });


    }
    });














    draft saved

    draft discarded


















    StackExchange.ready(
    function () {
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f48912%2fhow-to-validate-recommender-model-in-healthcare%23new-answer', 'question_page');
    }
    );

    Post as a guest















    Required, but never shown

























    1 Answer
    1






    active

    oldest

    votes








    1 Answer
    1






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes









    0












    $begingroup$

    You should still be able to use a validation set to evaluate the model, whether or not you pursue an experimental approach. (Specific features of your model and investigations may tweak these, but this is based on what's already been posted alone).



    There is nothing wrong with A/B group assignment and testing in a medical context, with a few caveats (this list is not exhaustive):




    • The relevant clinical/medical knowledge must be in a state of
      equipoise (it's not already clear that one approach is better than
      another, or which is better is genuinely not known).

    • Individuals should be aware that they are participating in a study, and that they are being routed to
      group A or B, and have the option to decline their assignment (or,
      conversely, they have been made aware of the experimental assignment
      and have consented to participate in advance).

    • An institutional review board should evaluate your proposed
      experiment and signed off on it. This, of course, presupposes that
      you have access to such a board composed of members able to make
      those assessments.


    Those can be a tall order, but you don't necessarily have to perform a prospective, double-blind experimental study in order to glean some information. A retrospective study could provide some insight as well, and your process for the validation set would be something like:




    1. Prepare your recommender model

    2. Feed your data through the model, without looking at actual outcomes

    3. Match your model output to actual outcomes to see whether or not
      people followed the recommendation (whether they ever saw that
      recommendation or not)

    4. Compare the results of people that ended up going with each
      recommended approach (A vs. B), as well as those who "followed" the
      recommendations or not (Recommended-A-did-A vs. Recommended-A-did-B,
      etc.)


    Retrospective studies are generally not as good as well-designed, well-executed prospective experimental studies, but they can still provide a lot of information. In situations where prospective experimentation is impossible or undesirable, the information a retrospective study provides may be the very best you can actually get.






    share|improve this answer









    $endgroup$


















      0












      $begingroup$

      You should still be able to use a validation set to evaluate the model, whether or not you pursue an experimental approach. (Specific features of your model and investigations may tweak these, but this is based on what's already been posted alone).



      There is nothing wrong with A/B group assignment and testing in a medical context, with a few caveats (this list is not exhaustive):




      • The relevant clinical/medical knowledge must be in a state of
        equipoise (it's not already clear that one approach is better than
        another, or which is better is genuinely not known).

      • Individuals should be aware that they are participating in a study, and that they are being routed to
        group A or B, and have the option to decline their assignment (or,
        conversely, they have been made aware of the experimental assignment
        and have consented to participate in advance).

      • An institutional review board should evaluate your proposed
        experiment and signed off on it. This, of course, presupposes that
        you have access to such a board composed of members able to make
        those assessments.


      Those can be a tall order, but you don't necessarily have to perform a prospective, double-blind experimental study in order to glean some information. A retrospective study could provide some insight as well, and your process for the validation set would be something like:




      1. Prepare your recommender model

      2. Feed your data through the model, without looking at actual outcomes

      3. Match your model output to actual outcomes to see whether or not
        people followed the recommendation (whether they ever saw that
        recommendation or not)

      4. Compare the results of people that ended up going with each
        recommended approach (A vs. B), as well as those who "followed" the
        recommendations or not (Recommended-A-did-A vs. Recommended-A-did-B,
        etc.)


      Retrospective studies are generally not as good as well-designed, well-executed prospective experimental studies, but they can still provide a lot of information. In situations where prospective experimentation is impossible or undesirable, the information a retrospective study provides may be the very best you can actually get.






      share|improve this answer









      $endgroup$
















        0












        0








        0





        $begingroup$

        You should still be able to use a validation set to evaluate the model, whether or not you pursue an experimental approach. (Specific features of your model and investigations may tweak these, but this is based on what's already been posted alone).



        There is nothing wrong with A/B group assignment and testing in a medical context, with a few caveats (this list is not exhaustive):




        • The relevant clinical/medical knowledge must be in a state of
          equipoise (it's not already clear that one approach is better than
          another, or which is better is genuinely not known).

        • Individuals should be aware that they are participating in a study, and that they are being routed to
          group A or B, and have the option to decline their assignment (or,
          conversely, they have been made aware of the experimental assignment
          and have consented to participate in advance).

        • An institutional review board should evaluate your proposed
          experiment and signed off on it. This, of course, presupposes that
          you have access to such a board composed of members able to make
          those assessments.


        Those can be a tall order, but you don't necessarily have to perform a prospective, double-blind experimental study in order to glean some information. A retrospective study could provide some insight as well, and your process for the validation set would be something like:




        1. Prepare your recommender model

        2. Feed your data through the model, without looking at actual outcomes

        3. Match your model output to actual outcomes to see whether or not
          people followed the recommendation (whether they ever saw that
          recommendation or not)

        4. Compare the results of people that ended up going with each
          recommended approach (A vs. B), as well as those who "followed" the
          recommendations or not (Recommended-A-did-A vs. Recommended-A-did-B,
          etc.)


        Retrospective studies are generally not as good as well-designed, well-executed prospective experimental studies, but they can still provide a lot of information. In situations where prospective experimentation is impossible or undesirable, the information a retrospective study provides may be the very best you can actually get.






        share|improve this answer









        $endgroup$



        You should still be able to use a validation set to evaluate the model, whether or not you pursue an experimental approach. (Specific features of your model and investigations may tweak these, but this is based on what's already been posted alone).



        There is nothing wrong with A/B group assignment and testing in a medical context, with a few caveats (this list is not exhaustive):




        • The relevant clinical/medical knowledge must be in a state of
          equipoise (it's not already clear that one approach is better than
          another, or which is better is genuinely not known).

        • Individuals should be aware that they are participating in a study, and that they are being routed to
          group A or B, and have the option to decline their assignment (or,
          conversely, they have been made aware of the experimental assignment
          and have consented to participate in advance).

        • An institutional review board should evaluate your proposed
          experiment and signed off on it. This, of course, presupposes that
          you have access to such a board composed of members able to make
          those assessments.


        Those can be a tall order, but you don't necessarily have to perform a prospective, double-blind experimental study in order to glean some information. A retrospective study could provide some insight as well, and your process for the validation set would be something like:




        1. Prepare your recommender model

        2. Feed your data through the model, without looking at actual outcomes

        3. Match your model output to actual outcomes to see whether or not
          people followed the recommendation (whether they ever saw that
          recommendation or not)

        4. Compare the results of people that ended up going with each
          recommended approach (A vs. B), as well as those who "followed" the
          recommendations or not (Recommended-A-did-A vs. Recommended-A-did-B,
          etc.)


        Retrospective studies are generally not as good as well-designed, well-executed prospective experimental studies, but they can still provide a lot of information. In situations where prospective experimentation is impossible or undesirable, the information a retrospective study provides may be the very best you can actually get.







        share|improve this answer












        share|improve this answer



        share|improve this answer










        answered 5 hours ago









        Upper_CaseUpper_Case

        1563




        1563






























            draft saved

            draft discarded




















































            Thanks for contributing an answer to Data Science Stack Exchange!


            • Please be sure to answer the question. Provide details and share your research!

            But avoid …



            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.


            Use MathJax to format equations. MathJax reference.


            To learn more, see our tips on writing great answers.




            draft saved


            draft discarded














            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fdatascience.stackexchange.com%2fquestions%2f48912%2fhow-to-validate-recommender-model-in-healthcare%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown





















































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown

































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown







            -

            Popular posts from this blog

            Ponta tanko

            Image
            Usona ponta tanko M60A1 Ponta tanko estas, simile kiel kirasita savveturilo, inter specialaj kirasitaj transportiloj. Kiel pontaj tankoj eblas konsideri ĉiujn memmovigaj rimedoj ebligantaj superi obstaklojn ĝis larĝeco 24 metroj rekte en spaco de bataloperacoj. Ties ĉefa tasko estas ebligi al la propraj taĉmentoj daŭrigi en rapida antaŭenirado sen senbezona reteniĝo. El tio pli-malpli rezultas taktikaj-teknikaj postuloj, kiujn ili devas plenumi. Precipe temas pri kapablo operacii komune kun unue-sinsekvaj taĉmentoj kaj transpontigi obstaklojn dum kelke da minutoj. This page is only for reference, If you need detailed information, please check here
            Read more

            Tantalo (mitologio)

            Image
            Ĉi tiu artikolo temas pri mitologio. Por aliaj signifoj vidu la paĝon Tantalo (apartigilo) . Tantalo , laŭ Gioacchino Assereto. Laŭ la helena mitologio, Tantalo (el la greka Τάνταλος , ŝancelanta aŭ tre malfeliĉa ) estis filo de Zeŭso kaj la oceanidino Pluto, laŭ la plej divastigata versio. Li estis reĝo de Frigio [1] aŭ de la monto Sipilo, en Lidio (Malgrandazio) [2] kaj post sia morto li suferis eternan torturadon ĉe la Tartaro, la plej profunda parto de Hadeso, rezervita por la plej perversaj homoj. Tamen, kelkaj rakontoj asertas ke li estis reĝo de Argo aŭ de Korinto [3] , aŭ ke li migris norden el la monto Sipilo por akiri la tronon de Paflagonio, el kie Ilo forpelis lin pro tio, ke Tantalo forkaptis kaj seksperfortis lian fraton Ganimedon [4] , kaj tiel ofendis la diojn. Verŝajne li ankaŭ regis urbon en okcidenta Anatolio, kiun oni nomis Tantalis [5] pro li, aŭ Sipilo , pro la monto sur kies flanko ankoraŭ staras ĝiaj ruinaĵoj [6] . Laŭ la geografiisto Strabo,
            Read more

            Erzsébet Schaár

            Image
            Erzsébet Schaár Memportreto de Erzsébet Schaár en Pécs Personaj informoj Naskiĝo 27-an de julio 1908 ( 1908-07-27 ) en Budafok Morto 29-an de aŭgusto 1975 ( 1975-08-29 ) (67-jara) en Budapeŝto Ŝtataneco Hungario Profesio skulptisto Familianoj Edz(in)o Tibor Vilt v   •   d   •   r Erzsébet Schaár [erĵEbet ŝAr], laŭ hungarlingve kutima sinsekvo Schaár Erzsébet estis hungara skulptisto. Ŝia edzo estis Tibor Vilt. Déryné Széppataki Róza en Miskolc fare de Erzsébet Schaár Erzsébet Schaár [1] naskiĝis la 27-an de julio 1908 en Budafok, ŝi mortis la 29-an de aŭgusto 1975 en Budapeŝto. Enhavo 1 Biografio 2 Elektitaj solaj ekspozicioj 3 Elektitaj kolektivaj ekspozicioj 4 Elektitaj skulptaĵoj 5 Elektitaj disĉiploj 6 Fontoj 7 Referencoj Biografio | Erzsébet Schaár studis en Industriarta Lernejo, poste en Reĝa Belarta Altlernejo (1924-1926). Ŝia ĉefinstruisto estis Zsig
            Read more